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Abstract. This note proves a weak type of the sharpness principle as con- 
jectured by Gneiting, Balabdaoui, and Raftery [9] in connection with proba- 
bihstic forecasting subject to caUbration constraints. A strong version of such 
a principle still awaits a proper formulation. 



1. Introduction 

This note is concerned with the conjectured sharpness principle described in 
the Gneiting-Balabdaoui-Raftery article on probabilistic forecasts [9] subject to 
calibration constraints. Using predictive distributions for making forecasts has 
been steadily gaining acceptance in various fields such as meteorology (Gneiting 
& Raftery [10]), economics (Granger [12]), and finance (Duffie & Pan [3]). The 
conceptual appeal of probabilistic forecasts and the advances in Markov Chain 
Monte Carlo methodology have made them very attractive. For a recent editorial 
on probabihstic forecasting, please see the article by Gneiting [8]. 

Predictive performances of forecasts have been traditionally measured by what 
is known as the probability integral transform or PIT. The PIT essentially involves 
applying the forecasted distribution functions to their corresponding true observa- 
tions and checking the uniformity of the resulting histogram. This method first 
proposed by Dawid |4j and further by Diebold, Gunther, & Tay [7], only leads to a 
necessary condition for a forecaster to be ideal. In fact, that using the PIT just by 
itself can lead to erroneous conclusions was succinctly demonstrated by an example 
of Hamill [13]. To address this issue Gneiting et al. in [9] introduce the concept 
of maximizing the sharpness of the predictive distributions subject to calibrations. 
They define calibration as 'the statistical consistency between the predictive distri- 
butions and the associated observations, and is a joint property of the predictions 
and the values that materialize'. Sharpness, on the other hand, 'refers to the con- 
centration of the predictive distributions and is a property of the forecasts only'. 
The authors then define a theoretical framework to assess calibration and sharpness 
and distinguish between several modes of calibration. Our article is based on this 
framework. 

Let {Ft, t = 1,2,...} and {Gt, i=l,2,...} denote sequences of continuous and 
strictly increasing distribution functions on the real line, possibly depending on sto- 
chastic parameters. One thinks of the Gt's as the true data-generating distributions 
and the corresponding sequence of FtS as the sequence of probabilistic forecasts. 
Among all the modes of calibration proposed in [9] , probably the most useful notion 
is that of probabilistic calibration. The sequence [Ft] is probabilistically calibrated 
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relative to the sequence {Gt} if 

1 ^ 

(1) ]im -Y,G^oF-\p)=p, VpG(0,l). 

i=l 

It is not difficult to see that probabilistic calibration is essentially equivalent to 
the uniformity of the PIT histogram, a fact that leads to the practicality of its 
applications. 

We should mention that calibration in the context of forecasting binary events 
is well studied in Statistics literature. See, for example, the articles by DeGroot 
& Fienberg [3], Dawid & Vovk [5], and Shafer & Vovk [T3]. The elegant game- 
theoretic approach of Vovk & Shafer [15] is a natural culmination of this approach. 
The models considered here, however, are continuous and not within the purview 
of the previous approach. 

2. Main results 

The particular problem addressed in this note is the one stated in Section 2.4 
of [S]. It involves what the authors of [5] call the sharpness principle. Ideally one 
would like to forecast the data generating distribution perfectly. Supposing that 
one maximizes sharpness among predictive distributions subject to calibration, how 
close to perfect forecasts can she get ? The strong sharpness principle in [9 states 
that perfect forecasts and maximization of sharpness subject to calibration is indeed 
the same. For probabilistic calibration it is difficult to find natural examples where 
such a principle holds. Thus, in [S], the authors also formulate the weak sharpness 
principle. Suppose the sequence {Ft} is probabilistically calibrated with respect 
to {Gt}. If one assesses sharpness by variance, what can one deduce about the 
accuracy of the forecasts Ftsl A partial answer, in the case of climatological 
forecaster (when all the FtS are the same), is provided in Theorem 1 of [9]. It 
is shown that on average the forecasts will have a bigger variance than the actual 
observations. A similar conclusion for the general case is conjectured, and this is 
what we settle here. The choice of averaged variance as a measure of sharpness 
is somewhat arbitrary. Please see section 3 in Gneiting et al. [TT] for further 
discussion. 

The main result is the following proposition. 

Proposition 1. Suppose, for some T £ N, the sequence of probability distributions 
{Fi} and {Gi} satisfy the finite probabilistic calibration condition 

I ^ 

(2) _^G'toFr^p)=p, for allp e {0,1). 

1 

For all such sequences {Fi, F2, ■ ■ ■ , Ft) and (Gi, G2, . . . , Gt) satisfying we 
have 

T T 

(3) ^Y.^ar{F,)>^Y.^ar{G.), 

1 1 

with equality if and only if 

(4) E{Gi) - E{Fi) = E{G2) - E{F2) = ... = E{Gt) - E{Ft). 

Here E{H) and Var{H) denote the expectation and the variance of a random vari- 
able with distribution function H. 
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With stronger assumptions one can extend the finite horizon result to a hmiting 
result consistent with the definition of probabilistic calibration given in [9]. Below 
we outline one such approach. Our assumptions are deliberately kept stronger than 
required in order to maintain simplicity of the statement. 

Proposition 2. Assume that there is a bounded interval (a, b) such that each Fi 
is zero to the left of a and one to the right ofb. Further assume that the sequences 
{Fi} and {Gi} satisfy the probabilistic calibration condition ([1]) and that there is a 
function : (0, 1) — > R such that 



Here lim sup and liminf denote the limit superior and the limit inferior of a real 
valued sequence. 

The significance of the previous inequalities is directly linked to the Gneiting- 
Balabdaoui-Raftery 9j paradigm of maximizing sharpness of a forecast subject to 
calibration. In practice, competing probabilistic forecasters need to be assessed on 
the basis of their predictive distributions and the outcomes that materialize, in ac- 
cordance with Dawid's [4] prequential principle. Ideally we would like a forecaster 
to predict the true distribution, i.e. Ft — Gt for all times t. However, Gt is unob- 
servable, so this is not an operational criterion. Instead [9] proposes maximizing 
sharpness of the predictive distributions subject to calibration. Our results lend 
further support to their proposal, in the sense that if we measure sharpness by 
average variance, a probabilistically calibrated forecaster will not be sharper than 
the ideal forecaster. 

However, Proposition[l] allows equality in ([3]) under apparently weaker conditions 
(jlj than distributional equality. The quest for notions of calibration that lead to 
the ideal forecaster as the unique sharpest forecaster (a strong form of the sharpness 
principle) remains open. 

Let us add that one should compare these results to existing results about de- 
composition of strictly proper scores into sharpness and reliability like terms for 
both binary forecasting and more general set-up. Please see the recent article by 
Brocker [2] for a complete result and references of previous work. Such a decompo- 
sition, of course, will be the most complete version of a sharpness principle that one 
can desire. However, in our framework, it is not obvious how to obtain a similar 
decomposition theorem. 

The rest of this note contains the proofs of the stated results. 



Proof of Theorem^ Let Z = {Zi, Z2, ■ ■ ■ , Zt) be a multinomial random vector 
with an equal probability of success in each of the T bins. Let U be the random 
variable 




Then 



(6) 




3. Proofs 



U := Fi{Xiy'F2{X2r' . ..Ft{Xt) 
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where X ~ [Xi^ X2, . ■ . ^ Xt) is a random vector such that margmaUy Xi ~ 
Gi, . . . , Xt Gt and is independent of Z. Then, as was mentioned in the pa- 
per [3], the finite probabihstic cahbration condition imphes [/ is a Uni(0, 1) random 
variable. 

For (u, z) among the possible values of ([/, Z), define the function 

(7) Hiz,u) = iFl^F^^...F^n''iu). 

We are going to compute Var{H{Z, U)) in two different methods. 

The first method is to condition on Z = z, and use the decomposition 

(8) Var{H{Z, U)) = E Var{H\Z = z) + Var E{H\Z = z). 

Now, it is straightforward to see that 

Var {H\Z = z) ^ Var{Xi), if z, = l,z.j = Q j 
E{H\Z = z) = E{Xi), if = l,Zj = Oj^i. 



(9) 
Thus 



T T 

Var{H{Z, U)) ^ + ^ - fi? ■ 



^ i=l i=l 



Here fii = E{Gi) and /i is the mean of the /i^'s. Thus 

1 ^ 

(10) Var{H{Z,U))>-Y,yo^r{G,) 

1=1 

with equality if and only if all the Gi's have the same mean. 

The second method to compute the variance of H{Z, U) is to first condition on 
U , and repeat the variance decomposition formula 

(11) Var{H{Z, U)) = E Var{H\U ^ p) + Var E{H\U = p). 
To compute the above, note that 

E{H\U = p) = ^Y.Fr\p)^ E{H^\U = p) = 1 (^;-l(p))^ 

(12) 

Var{H\U=p) = }^Y.{Fr\p)f 1^ (Ei^r'b) ) ■ 

Define Yi = F~^{U), i = 1,2, ... ,T. By the finite probabilistic calibration condi- 
tion, U is marginally uniform. Hence, each Yi has distribution Fi. Now, from the 
equalities in (fTTj) and (fT2|) . we get 

Var{H{Z, C/)) = ^ {Y^ ~ (j^ y) 

i=l \i=l J 

(13) +^y-r{f:Y}j 
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The finally equality follows from the fact that for any Z, one has Var{Z) = EZ^ 
{EZf. 

Thus, we get the equality 

2 



T T / ^ \ 

Var{H{Z, U)) ^ ^Jl ^«^(^^) + ^ E(^^^)' " ^ E ^(^^) ' 

i=l 1 \i=l / 
T T 



where tti = E(Fi) and a is the mean of the a^'s. 

Thus, if all ai's (i.e., all the means of Fis) are the same, we get exact equality 
above. In that case, combining with inequality (fTO| . we get the result 



T T 

(15) ^Y.^ar{F,)>^Y.^ar{G, 



with equality if and only if the means of all the G^'s are the same. 
To prove ([3]), in case when all the a,;'s are not the same, define 

(16) F*{x)=F,{x + ai), G*{x)^Gr{x + a^). 

Notice, that the calibration condition ^ continues to hold. Additionally, each F* 
has now mean zero and the same variance as Fi. Applying the variance inequality 
([T5)) for case of {F*} and {G*}, we get the complete result as in the statement. □ 

Proof of Theorem [2l The proof proceeds by extending the idea of variance decom- 
position in the finite case to the limit as T tends to infinity. For every fixed T, as 
before, one defines 

(17) U{T) := FiiX^f' F2iX2f' . . . FriXrf^ 

where X = {Xi, X2, ■ ■ ■ , Xt) is a random vector such that marginally Xi ~ 
Gi, . . . , Xt ^ Gt and is independent of Z{T) = (Zi, . . . , Zt), which is a multino- 
mial with T cells and equal probability of success in each cell. 

Now, the difference here with the finite case is that U{T) is no longer uniform. 
However, it follows from the condition ([T]) that U{T) converges in distribution (or, 
weak convergence, see Billingsley [T]), as T tends to infinity to J7, which is a Uni(0, 1) 
random variable. However, one can still proceed with a variance decomposition as 
in the last proof. Notice that equality pH)) continues to hold for each finite T when 
we define H in d?]) by replacing Z by Z{T) and U by U{T). 

The equality in (fT4)) also holds, but Yt does not follow Fi since U (T) is not 
uniform. However, from (|13p . one can still write the following. 



Var{H{Z{T),U{T))) < E {Fr\u {T)Y 

1=1 

Combining the observations above, we get that for every fixed T, one has 

T 



(18) E 



l5:(Fr^(f/(r))^ 



1 ^ 
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Now U{T) converges to U in distribution. Define the sequence of functions 

1=1 

Then, each 6t is continuous and bounded by our assumptions. Also, as T tends 
to infinity, 9t converges to 6 uniformly. Thus, from the standard theory of weak 
convergence, it follows that limj-^oo E9t{U {T)) exists and is given by E9{U) where 
U is Uni(0, 1). In particular, from HH]), it follows that 

1 ^ 

(19) E9{U) > limsup - V Var{Gi). 

Assume, for now, that each Fi has the same zero mean. Since 9t converges to 9 
pointwise and remain uniformly bounded, it follows from the Dominated Conver- 
gence Theorem that 

1 ^ 

(20) E9iU) = hm E9t{U) - - ^ VariF,), 

1=1 

since F^^{U) has the distribution Fi and we have assumed its mean to be zero. 
Combining and (|20p wc have proved the stated proposition. 

Finally, the mean zero assumption on F^'s can be easily removed by suitable 
translations as was shown in the proof of the last proposition. □ 
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